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Preface 


This  report  describes  an  effort  at  IBM  Research  developing  a  new  architecture  for  digital  packet  video 
systems  that  are  both  power-and  bandwidth-efficient  in  support  of  visualization  applications  in  the  US 
Army,  including  systems  applicable  to  flexible  displays  for  the  soldiers  in  the  field.  Potential  candidates 
for  high-resolution  visualization  applications  are  for  example  video  conferencing,  surveillance  and 
unmanned  vehicles  which  all  have  to  work  within  given  bandwidth  constraints  that  can  vary  with  time. 
The  hardware  development  for  this  reconfigurable  and  scalable  prototype  platform,  called  BlueEagle,  was 
completed  during  the  period  from  September  2002  to  November  2004,  contract  number  DAAD 19-02-2- 
0023,  under  the  direction  of  the  US  Army  Soldier  Systems  Center  in  Natick,  MA  and  the  US  Army 
Research  Laboratory  in  Adelphi,  MD. 

After  the  contract  ended  on  November  30,  2004,  IBM  Research  continued  writing  embedded  code  for  this 
reconfigurable  platform  with  a  commitment  to  deliver  two  demo  systems  to  the  US  Army.  The  first 
system  was  delivered  on  June  7  and  8,  2005,  to  the  ARL  site  in  Adelphi,  MD.  The  second  system  was 
delivered  on  July  11,  2005,  to  the  NSC  site  in  Natick,  MA.  Between  the  installations,  firmware  and 
software  updates  occurred  to  keep  both  systems  on  the  same  level.  There  will  be  further  updates  to  the 
system  until  the  end  of  2005. 
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A  NEW  ARCHITECTURE  FOR  DIGITAL  PACKET  VIDEO  SYSTEMS 
THAT  ARE  POWER-  AND  BANDWIDTH-EFFICIENT 

SUMMARY 

The  focus  of  this  effort  has  been  to  build  a  modular,  scalable  and  reconfigurable  platform  to  test  new 
architectures  for  digital  packet  video  systems  with  an  emphasis  on  bandwidth  conservation  along  with 
low  power  consumption.  In  many  real  environments  the  bandwidth  is  dictated  by  the  network's  abilities 
and  it  cannot  change  without  huge  investments  in  upgrading  the  infrastructure.  Efficient  and  effective 
utilization  of  available  bandwidth  and  power  is  one  of  the  biggest  obstacles  for  getting  meaningful  visual 
information  into  the  user's  hands  in  order  to  make  timely  and  correct  decisions  on  high-information 
content  imagery.  The  project  was  nicknamed  “BlueEagle”  since  one  of  the  applications  is  ultra-high- 
resolution  video  surveillance. 

An  early  decision  was  made  to  make  the  “BlueEagle”  system  modular.  This  was  done  both  to  keep 
the  task  of  designing  this  hardware  a  manageable  effort  and  to  allow  flexibility  for  interconnection  of 
“BlueEagle”  system  components.  Furthermore,  each  of  the  modules  can  serve  other  purposes. 

The  scalability  of  the  “BlueEagle”  system  derives  from  its  modularity  and  is  essential  for  the 
handling  of  ultra-high-resolution  video  content.  Each  module  of  the  “BlueEagle”  system  is  designed  to 
handle  at  least  1920x1080  pixels  at  30Hz  frame  video  rate  progressively.  Connecting  multiple  modules 
together  allows  scalability  to  3840x2160  pixels  at  60Hz  video  frame  rate  and  beyond. 

The  third  main  characteristic  of  our  “BlueEagle”  system  is  reconfigurability.  This  allowed  us  to 
design  the  overall  system  intelligently,  using  the  same  physical  modules  on  both  sides  of  the  network  and 
thus  reducing  the  number  of  different  module  designs  from  six  down  to  three.  Nevertheless,  the 
complexity  of  each  module  resembles  that  of  a  PC  motherboard.  Completing  these  complex  designs  with 
a  small  group  of  people  at  IBM  Research  was  a  challenging  task  because  many  different  technical 
disciplines  were  required.  Examples  include  knowledge  of  compression  algorithms,  multi-type  memory 
controllers,  video  interfaces,  networking  and  cutting-edge  reconfigurable  chips  (FPGAs)  with  embedded 
PowerPCs.  We  used  the  latest  FPGAs  available  which  are  best  suited  for  video  processing,  even  though 
these  chips  were  available  only  as  engineering  samples  throughout  the  first  half  of  the  project.  The 
complexity  of  these  leading-edge  FPGAs  paid  off  in  the  design  as  we  were  able  to  design  each  module  of 
the  “BlueEagle”  project  with  a  single  FPGA  in  1 152-pin  or  1536-pin  BGA  package. 

The  “BlueEagle”  System  consists  of  three  different  modules:  the  Input/Output  Module  (IOM),  the 
Image  Processing  Module  (IPM)  and  the  Network  Interface  Module  (NIM).  Communication  between  the 
modules  is  established  by  high  speed  differential  serial  link  provided  by  the  Rocket-IO  connections  of  the 
FPGAs,  utilizing  the  Aurora  protocol  provided  by  Xilinx.  Infiniband  protocol  is  also  possible  (in  case 
connection  to  other  equipment  is  needed)  because  we  chose  Infiniband  connectors  and  impedances  to 
establish  the  physical  connection  between  the  modules. 

The  Input/Output  Module  (IOM)  is  used  in  two  places:  as  input  interface  module  to  the  camera 
system  and  as  output  interface  module  to  a  high-resolution  visual  system.  The  camera  system  consists  of  a 
CMOS  or  CCD  block  camera  with  pan&tilt  unit  and  motorized  lens.  It  was  requested  that  the 
“BlueEagle”  system  be  able  to  connect  to  different  video  camera  interface  types.  Therefore  we  allow 
video  inputs  from  HD-SDI  (SMPTE  292M  Standard),  DVI-D  (DVI  1.0  Standard)  and  CameraLinkTM 
(Camera  Link  SpecficiationsVl.l).  A  video  data  rate  up  to  3.96Gbit/sec  can  be  ingested  into  each  input 
module  of  the  “BlueEagle”  system.  The  high-resolution  visual  system  consists  of  a  DVI-enabled  monitor, 
keyboard,  mouse  and  overlaying  GUI  from  a  supporting  workstation.  In  addition  to  the  interface  adapter 
functions,  the  IOM’s  purposes  are  packetization/depacketization  of  the  image  data  into  background  and 
regions  of  interest  (ROIs)  as  well  as  background  filtering. 
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The  Image  Processing  Module  (IPM)  is  responsible  for  encoding  the  packetized  image  data  on  the 
transmitting  side  and  on  the  receiving  side  to  decode  the  data  stream.  In  the  first  few  months  in  2003  we 
chose  JPEG2000  as  our  preferred  compression.  As  compared  to  alternative  choices  for  this  application, 
the  JPEG2000  algorithm  is  superior  and  more  scalable  with  increased  image  size.  Compared  to  baseline 
JPEG,  JPEG2000  improves  the  compression  efficiency,  allows  lossy  and  lossless  compression,  allows  the 
presentation  of  multiple  resolutions,  allows  tiling,  allows  region  of  interest  coding,  allows  bit  depth  higher 
than  8bit  and  good  error  resilience.  The  design  of  the  of  the  IPM  was  the  most  complex  module  in  terms 
of  printed  circuit  board  routing  because  we  decided  to  use  the  largest  FPGA  available,  together  with 
DDR  memory  and  QDR  memory.  The  FPGA  is  paired  with  a  TI  DSP  to  allow  some  diversion  of  FPGA 
functionality  to  the  DSP,  and  to  free  up  resources  in  the  FPGA.  This  module  has  a  very  high  serial  IO 
bandwidth  with  16  links,  each  with  a  maximum  data  rate  of  3.125Gbit/sec  (on  production  grade  FPGAs). 

The  Network  Interface  Module  (NIM)  provides  connection  to  10/100/1000  Mbit-Ethemet  networks, 
two  DVI  outputs,  two  Fibrechannel  ports  for  potential  storage,  full  data  rate  capable  IEEE 13  94,  serial  PC 
interfaces  and  a  smartcard  interface  for  authentication  implementations.  This  module  has  a  large  amount 
of  DDR  memory  for  multi-frame  storage,  allowing  compensation  for  variations  in  network  data  rates. 

Through  November  2004  of  this  project,  the  three  modules  (IOM,  IPM  and  NIM)  were  designed, 
fabricated  in  small  quantities,  and  successfully  tested  separately  for  their  functionality  before  developing 
embedded  code  for  the  overall  platform.  A  “Roving  Eye”  application  was  developed  and  demonstrated  to 
show  the  capabilities  of  the  system. 

IBM  developed  software  for  an  overlaying  GUI  (1920x1200  pixels),  designed  to  give  the  user  overall 
control  of  the  system  by  implementing  a  1920x120  pixels  toolbar.  The  toolbar  can  be  controlled 
preferably  by  a  trackball  or  by  a  mouse  and  it  allows  full  control  of  the  remote  camera  system,  control  of 
size  of  the  region  of  interest  and  also  control  of  spatial  and  temporal  parameters  of  the  background.  In 
addition,  the  software  can  establish  a  link  to  the  PeopleVision  Software  from  IBM  Research  which 
enables  automatic  tracking  of  persons  and  specially-defined  objects.  When  linked  to  the  PeopleVision 
Software,  the  GUI  software  receives  the  region(s)  of  interest  and  allows  the  “BlueEagle”  system  to  apply 
more  bandwidth  to  that  area. 

Development  of  the  embedded  code  is  complex,  and  we  have  continued  our  development  beyond 
November  30,  2004.  We  delivered  two  “BlueEagle”  systems  for  demo  purposes  to  the  Army  Research 
Laboratory  and  Natick  Soldier  Center  in  July  2005.  Thereafter,  the  system  could  be  scaled  up  to  4x 
HDTV  resolution  (3840x2160  pixels)  or  modules  of  the  system  can  be  used  for  other  purposes.  Examples 
are: 

IOM:  Generic  digital  video  capture  module  for  HD-SDI,  CameraLink  and  DVI 

IPM:  Hardware  implementation  of  any  compression  algorithm  in  real-time 

NIM:  Network  attached  desktop  implementation  (no  harddrive,  no  OS)  in  secure  environments, 

DPVL  multi-monitor  adapter  plateform  for  command&control  environment 

The  goal  of  this  project  is  to  use  “BlueEagle”  modules  as  a  prototype  platform  for  different  visual 
applications,  including  high-speed  wireless  links.  From  this  work  we  wish  to  derive  a  potential  ASIC 
chipset  solution,  which  would  reduce  component  cost  and  also  dramatically  reduce  packaging  size. 

Digital  high-resolution  video  surveillance  is  on  the  rise  because  of  availability  of  new  HD  CMOS  sensors 
at  lower  prices.  It  is  expected  that  within  the  next  5  years,  commercial  video  surveillance  will  be 
predominated  by  digital  technology  at  HD  level  resolution.  Therefore,  video  surveillance  in  the  defense 
sector  will  be  going  beyond  HD  resolutions,  such  as  4k  or  8k  horizontal  resolution.  In  the  entertainment 
side  of  the  high-end  commercial  sector,  the  Digital  Cinema  Initiative  (DCI)  December  2004  preliminary 
specifications  embrace  JPEG2000  as  their  compression  algorithm  because  of  its  bit  depth  beyond  8bit  and 
frame-by-frame  handling  for  studio  editors.  Taken  together,  these  developments  boost  the  likelihood  of  a 
highly-integrated  “BlueEagle”  solution,  compromising  all  the  capabilities  into  a  single  chip. 
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1.  Introduction 


Transmission  of  high-resolution  video  images  over  communication  channels  requires  a  large 
amount  of  channel  bandwidth.  The  large  bandwidth  requirements  are  a  result  of  high  data  rates 
required  to  communicate  high-resolution  video  data,  particularly  as  the  video  image  information 
content  is  increased  from  640x480  (NTSC)  to  1920x1080  (HDTV)  and  beyond,  needed  for  many 
applications.  Digital  video  cameras  today  can  capture  images  more  than  four  times  larger  than 
HDTV.  The  large  bandwidth  requirement  is  imperative  unless  techniques  such  as  compression  are 
applied  to  the  video  data.  The  problem  of  communicating  video  data  signals  is  further  compounded 
in  situations  where  available  channel  bandwidths  differ  at  various  stages  in  the  capture,  transmission 
and  reconstruction  of  the  video  data,  or  where  available  channel  bandwidth  may  be  changing  over 
time.  For  portable  imaging  applications,  power  consumption  is  a  major  issue.  Power  consumption  in 
video  imaging  systems  depends  primarily  on  the  amount  of  processed  video  pixel  data,  and  applies  to 
all  portions  of  the  imaging  chain:  capture,  compression,  transmission,  and  reconstruction.  Hence, 
reducing  the  bandwidth  and  power  remains  an  important  objective  in  video  communications. 

A  common  approach  to  reducing  bandwidth  requirements  in  video  transmission  is  to  use  image 
compression.  Compression  techniques  typically  compress  entire  images,  with  compression  ratios  and 
resulting  image  quality  chosen  for  a  given  application  and  nature  of  image  content.  Compression 
methods  can  be  adapted  to  respond  to  changes  in  image  aspects  but  they  cannot  be  considered 
intelligent  processing  of  the  video  image  in  comparison  to  the  visual  system  of  a  human  observer. 

In  the  human  visual  system,  a  large  number  of  photoreceptors  are  concentrated  in  the  foveal 
region  of  the  eye.  As  the  eyes  dart  between  various  regions  of  interest  in  the  image,  only  a  small 
region  of  the  image,  near  the  fixation  point,  is  processed  by  the  human  visual  system  to  contain  high- 
resolution  image  information.  Compression  techniques  process  the  entire  image  even  if  only  small 
regions  of  interest  need  to  contain  high-resolution  visual  data.  In  some  systems,  an  attempt  is  made  to 
enhance  the  image  content  by  creating  a  magnified  portion  of  the  image  within  the  overall  field  of 
view.  However,  the  magnified  region  necessarily  blocks  adjacent,  un-magnified  background  regions. 
For  surveillance  applications,  use  of  a  magnified  region  reduces  situational  awareness,  and  for  other 
imaging  applications,  use  of  a  magnified  region  reduces  navigation  ability. 

Efforts  have  been  made  to  create  foveated  displays  and  supporting  systems  to  reduce  data 
rates.  These  systems  display  higher  image  information  content  within  a  region  of  interest,  the  location 
of  which  is  set  by  an  eye-tracking  device,  worn  by  the  user.  Typically,  a  filter  is  applied  to  the  video 
data  to  define  the  region  of  interest,  in  which  image  quality  falls  off  gradually  for  pixel  data  farther 
from  the  center  of  the  region  of  interest. 

Overall,  efforts  to  create  foveated  displays  have  remained  unsatisfactory  for  many 
applications.  If  the  distance  between  the  image  capture  device  and  display  is  large  enough  to  cause  a 
signal  delay  more  than  a  few  milliseconds,  the  utility  of  a  system  containing  an  eye-tracking  device  is 
very  limited.  Eye-tracking  devices  tend  to  be  cumbersome,  expensive,  and  require  calibration  for 
good  performance.  Conventional  foveated  video  imaging  systems  do  not  provide  real-time  controls  to 
allocate  available  bandwidth  to  various  portions  of  the  video  image  data,  or  adapt  to  changes  in  the 
available  bandwidth.  These  systems  do  not  provide  independent  control  of  the  resolution  of  portions 
of  the  video  data,  namely  spatial  sampling,  temporal  sampling,  compression,  and  color  bit-depth. 
Finally,  none  of  these  systems  contain  provisions  for  buffering  the  video  data,  with  partial-frame 
packetization,  encoding,  or  encryption  of  digital  video  data,  suitable  for  either  wireless  transmission 
or  transmission  over  a  network.  For  surveillance  applications,  it  is  important  to  have  a  seamless 
image  presentation  of  multiple  regions  of  interest,  with  user  or  other  software  control  for  monitoring 
and  tracking  targets. 

Hence,  there  is  a  need  for  a  video  communication  system  that  requires  relatively  low  bandwidth 
and  power,  through  selective  allocation  of  available  system  bandwidth  to  regions  of  interest  within 
the  video  data.  Such  a  system  would  need  to  be  able  to  provide  independent  control  of  the  resolution 
of  portions  of  the  video  data,  namely  spatial  sampling,  temporal  sampling,  compression,  and  color 
bit-depth. 
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2.  Architectural  Design  of  the  Scalable  “BlueEagle”  Platform 

2.1  System  architecture  in  manageable  functional  modules 

Right  from  the  beginning  of  this  project  it  was  decided  to  build  the  system  modular  in  order  to  use  the 
limited  resources  more  efficiently.  Figure  1  shows  the  “BlueEagle”  system  in  1920x1080  mode  setup 
as  it  was  delivered  to  two  sites  for  the  US  Army.  The  boxes  in  color  represent  the  different 
“BlueEagle”  system  modules.  The  same  color  means  that  physically,  the  hardware  is  identical  but 
contains  different  firmware  for  the  specific  functionality. 

Location  A 

Silicon  Imaging  High-Bandwidth  channels  to 


The  communication  between  the  modules  is  running  on  the  physical  Rocket-IO  layer  of  the  Xilinx 
FPGA  chips  using  the  Aurora  protocol.  The  interface  between  the  modules  carries  video  packet  as 
well  as  control  packets.  The  control  packets  are  necessary  for  system  control  and  pheripheral  control. 


2.2  Choice  of  prefered  compression  algorithm 

Essentially  the  system  is  a  reconfigurable  system  and  therefore  any  kind  of  compression  algorithm 
can  be  implemented  unless  it  exhausts  the  resources  of  the  hardware.  We  decided  during  the  course 
of  the  contract  to  choose  JPEG2000  as  our  choice  of  compression  because  it  is  a  wavelet-based 
algorithm  which  allows  defining  regions  of  interest.  Furthermore,  JPEG2000  allows  for  lossy  and 
lossless  modes.  JPEG2000  is  a  very  complex  compression  algorithm  and  we  limited  ourselves  to 
compress  only  the  region  of  interest  after  the  packetizer.  The  background  information  is  low 
bandwidth  anyway  and  gets  a  different  threatment. 

Figure  2  shows  the  first  implementation  of  the  encoder  which  fits  into  the  IPM  and  is  still  being 
debugged. 

We  have  also  plans  to  prototype  different  and  new  algorithms  for  JPEG  on  the  “Blue  Eagle”  hardware 
and  maybe  if  resources  allow  it  move  the  compression  into  the  IOM  if  we  populate  it  with  a  bigger 
FPGA. 
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Figure  2:  First  implementation  of  the  compression  on  the  encoder  side  in  the  IPM 


2.3  Scalability 


Figure  3  Work  is  ongoing  in  the  implementation  of  the  functionalities  for  each  of  the  six  modules 
(Input  IOM,  Encoding  IPM,  Transmitter  NIM,  Receiver  NIM,  Decoding  IPM  and  Output  IOM)  that 
will  be  used  for  the  single  channel  demonstration  system.  It  was  decided  to  postpone  integration  of 
the  Camera-Link  interfaced  camera  until  after  the  upcoming  demo  to  allow  focus  on  background 
image  filtering,  compression  and  overall  system  functionality. 
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Figure  3:  Scaled-up  “BlueEagle”  system  to  4x  HDTV  resolution 
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3.  “BlueEagle”  Hardware  Module  Design 

3.1  Input/Output  Module  (TOM) 

As  a  module  on  the  transmitting  side  of  the  “BlueEagle”  system,  the  IOM  needs  to  be  able  to 
connect  to  different  camera  interfaces  (HD-SDI,  DVI  and  CameraLink™)  and  to  accomplish  the 
preprocessing  of  the  video  data  (filtering  and  packetization). 

As  a  module  on  the  receiving  side  of  the  “BlueEagle”  system,  the  IOM  needs  to  do  video  post¬ 
processing  (de-packetize  the  video  data,  merge  the  right  packets  and  do  the  inverse  filtering)  and 
output  the  video  date  via  DVI  channel  on  a  monitor  or  projector. 

As  indicated  in  Section  2,  this  was  achieved  by  designing  a  module  which  can  accomplish  both 
functions  by  different  firmware  on  the  FPGA  but  identical  hardware. 

3.1.1  Input/Output  Module  (IOM)  Design 

Figure  4  shows  the  block  diagram  of  the  IOM  with  all  its  peripheral  interfaces  which  are  accessible 
from  the  outside.  On  the  left  is  the  data  input  for  the  various  video  camera  interfaces.  On  the  right 
side  the  outputs  for  HD-SDI  monitoring,  DVI  monitor  output  and  Aurora  channels  on  Infiniband  lx 
connectors  to  the  next  module,  the  IPM.  The  additional  Aurora  channels  on  the  top  are  for  the 
optional  scalability  of  the  system  to  expand  resolution  beyond  1920x1080  resolutions.  On  the  bottom 
of  Figure  4  are  all  the  generic  interfaces  of  which  some  are  use  differently  whether  the  module  is 
used  on  the  transmitting  or  receiving  side  of  the  “BlueEagle”  system  (see  Figure  5). 
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Figure  4:  Input/Output  Module  (IOM)  functional  block  diagram 
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In  case  of  a  HD-SDI  camera  with  an  interlaced  1920x1080  videostream  we  implemented  Gennum 
chipset  consisting  of  a  motion  Co-Processor  and  a  de-interlacer  chip  which  proved  to  give  us 
professional  quality  progressive  output  as  input  to  the  FPGA.  The  FPGA,  a  Xilinx  Virtex-IIPro  20 
chip,  is  the  centerpiece  of  the  board  controlling  all  other  peripheral  chips.  It  contains  two  embedded 
PowerPC  chips  which  are  to  be  used  as  a  microcontroller  for  the  unit  or  in  the  future  also  to  run 
embedded  Linux  on  it. 

Figure  5  shows  a  photograph  of  the  assembled  IOM  PCB  with  its  1,654  parts.  Altogether  there  are 
146  different  parts  on  the  board  and  the  PCB  has  an  overall  of  7,128  pins.  Ninety-eight  percent  of  all 
the  SMD  components  could  be  placed  automatically  by  our  pick&pace  tools  and  only  30  parts  had  to 
be  populated  by  hand  beside  all  the  through-hole  connectors.  The  board  is  16”  x  5.6”  in  size  and  is 
powered  by  standard  56 W  Thinkpad  certified  international  power  supplies. 


■  ■ 


Figure  5:  Photograph  of  the  fully  populated  Input/Output  Module  (IOM)  after  PCB  assembly 
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3.1.2  Input/Output  Module  (IOM)  Specifications 


Figure  6  shows  the  IOM  specifications  at  a  glance.  HD-SDI  is  compliant  to  SMPTE  292  and 
SMPTE  274.  It  has  been  programmed  to  use  the  HD-SDI  signal  from  the  Ikegami  HDL-40  camera 
which  is  1 920x1 080@60Hz  progressive.  The  DVI  input  and  output  is  able  to  go  up  to  165MHz  as 
specified  by  the  DVI  1.0  standard. 


Category 

Parameter 

Value 

Comment 

Inputs 

HD-SDI 

DVI-D 

CameraLinkTM 

1.495  Gb/s 

3.96  Gb/s  (max.  165  MHz  pixel 
CLK) 

2.38  Gb/s 

SMPTE  292M 

Compliant  to  DVI  1.0 

Outputs 

DVI-D 

HD-SDI 

Infiniband 

3.96  Gb/s  (max.  165  MHz  pixel 
CLK) 

1.495  Gb/s 

600  Mb/s  to  3.125  Gb/s 

Compliant  to  DVI  1.0 
SMPTE  292M 

RocketIO,  reconfigurable 

Control 

Interfaces 

RS232  &  Aux. 
USB1.1 

IEEE  13  94a 

115kb/s 

12Mb/s 

400Mb/s 

Others: 

Keyboard 

Mouse 

FPGA 

Embedded 

CPU 

Rocket-IO 

2x  Power  PC 

Aurora/Inifiniband 

Embedded  Linux  optional 

Infiniband  lx  connectors 

PC  Board 

16-layer 

16”  x  5.6”  x  0.093”, 
0.61kg/1.351b 

Weight  fully  populated 

Chassis 

0.045”  steel 

16.5”  x  5.75”  x  1.5”, 
1.54kg/3.401b 

Total  weight  2.15kg/4.751b 

Power 

Standby 

Typical 

-  8.7  W  (without  input  signal), 

-  19.8W  (with  input  signals) 

Fully  programmed 
Simultaneous  HD-SDI&DVI 

Memory 

Deinterlacer 
Frame  buffer 
PowerPC 

96  MByte  SDRAM 

96  MByte  SDRAM 

32  MByte  SDRAM 

Upgrade  by  2x  possible 

Ext. 

storage 

Compact  flash,  Microdrive 

Embedded  Linux  storage 

Figure  6:  Summary  table  of  specifications  of  the  Input/Output  Module  (IOM) 


During  the  project  we  used  exclusively  RS232  as  control  interfaces  due  to  easier  software 
implementation  even  though  USB  1.1  and  IEEE 13 94a  are  also  provided  by  the  hardware.  The 
modular  interconnection  is  done  via  Aurora  protocol  which  is  available  in  the  Infiniband  lx 
connectors.  The  choice  of  these  connectors  leaves  then  the  option  open  to  use  Infiniband  protocol  in 
the  future  if  required.  The  Gennum  deinterlacer  and  motion-coprocessor  have  their  own  memory 
aside  from  the  frame  memory  which  is  accessible  through  the  FPGA.  The  compact  flash  is  primarily 
used  to  store  boot-time  FPGA  configurations  and  they  can  easily  be  exchanged  for  new  firmware 
updates. 
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3.1.3  Input/Output  Module  (IOM)  Packaging 


In  order  to  protect  the  PCB  board  and  the  sensitive  electronics,  we  designed  a  metal  enclosure  out 
of  folded  steel  which  was  lasercut  precisely.  The  enclosure  also  guides  the  airflow  in  a  defined 
channel  limited  by  the  enclosure  and  therefore  improves  cooling  of  the  components  of  the  PCB  board, 
especially  the  Gennum  chipset,  the  FPGA  and  the  DC/DC  converters. 

Figure  7  and  Figure  8  show  photographs  of  the  packaged  Input/Output  modules  suitable  for  19 
inch  rack  mounting.  The  height  is  1U  or  1.75  inches.  All  the  camera  input  signals  are  accessible  in  the 
front  while  the  DVI  output,  the  four  Aurora  channels  and  the  compact  flash  are  in  the  back. 


Figure  7:  Front  view  of  IOM  19”-rackmountable  1U  module  with  digital  video  capture  interfaces 


Figure  8:  Back  view  of  IOM  19”-rackmountable  1U  module  (J5  through  J8  on  the  left  side  are  additional 

high  speed  channels) 
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3.2  Image-Processing  Module  (IPM) 


As  a  module  on  the  transmitting  side  of  the  “BlueEagle”  system,  the  IPM  is  taking  care  of  the  video 
encoding  and  on  the  receiving  side  it  needs  to  do  the  decoding  of  the  video  information.  It  was 
decided  that  we  dedicate  a  whole  separate  module  for  this  funtionality  due  to  the  complexity  of  the 
compression  algorithms  and  to  keep  the  functionalities  separate  for  easier  code  development. 

As  indicated  in  Section  2,  this  was  achieved  by  designing  the  IPM  such  that  it  can  accomplish  both 
functions  by  different  firmware  on  the  FPGA,  but  identical  hardware  on  both  the  transmitting  and 
receiving  side. 

3.2.1  Image  Processing  Module  (IPM)  Design 

Figure  shows  the  block  diagram  of  the  IPM.  It  has  been  designed  specifically  for  image  processing 
by  pairing  the  Xilinx  Virter-IIPro  70  FPGA  with  192Mbyte  of  DDR  and  9Mbyte  of  QDR  in  order  to 
have  high  bandwidth  available  to  memory.  In  the  IOM  we  used  SDRAM  only. 

In  addition  to  that  we  put  a  600MHz  TI  DSP  on  the  board  in  case  the  FGPA  will  get  overloaded  with 
tasks.  Main  interface  with  the  other  modules  is  the  Aurora  protocol  accessible  on  Infiniband  lx 
connectors.  In  this  board  we  also  implemented  the  first  time  the  chip  scope  connector  for  faster 
debugging  capabilities. 


Aux.  Aurora  / 
Infiniband  for 
Scalability 


Figure  9:  Image  Processing  Module  (IPM)  functional  block  diagram 


Figure  10  shows  a  photograph  of  the  assembled  IPM  PCB  with  its  1,790  parts.  Altogether  there  are 
144  different  parts  on  the  board  and  the  PCB  has  an  overall  of  10,101  pins.  Ninety-eight  percent  of 
all  the  SMD  components  could  be  placed  automatically  by  our  pick&pace  tools  and  only  36  parts  had 
to  be  populated  by  hand  beside  all  the  through-hole  connectors.  The  board  is  16”  x  5.6”  in  size  and  is 
powered  as  the  IOM  by  standard  56 W  Thinkpad  certified  international  power  supplies. 


10 


Figure  10:  Photograph  of  the  fully  populated  Image  Processing  Module  (IPM) 


3.2.2  Image-Processing  Module  (IPM)  Specifications 

A  summary  of  the  IPM  specifications  is  shown  in  Figure  11.  The  IPM’s  IO  capability  is  mainly 
based  on  Rocket-IO  from  the  FPGA  in  order  to  provide  high  bandwidth  (600Mbps  to  3.125Gbps  per 
link).  In  total  14  Rocket-IO  links  with  Aurora  protocol  are  available  on  the  Infiniband  4x  connector. 


Category 

Parameter 

Value 

Comment 

Inputs 

Infiniband 

600  Mb/s  to  3.125  Gb/s 

14  Possible  Links. 

Rocket-IO,  reconfigurable 

Outputs 

Infiniband 

DVI-D 

600  Mb/s  to  3.125  Gb/s 
3.96  Gb/s  (max  165  MHz 
pixel  CLK) 

14  Possible  Links. 

Rocket-IO,  reconfigurable 

Compliant  to  DVI  1.0 

FPGA 

Embedded 

CPU 

Rocket-IO 
lx  Power 

PC 

Aurora/Inifiniband 
Embedded  Linux  optional 

Used  Infinibadn  lx  connectors 

PC  Board 

16-layer 

16”  x  5.6”  x  0.093”, 
0.64kg/1.511b 

Weight  fully  populated 

Chassis 

0.045”  steel 

16.5”  x  5.75”  x  1.5”, 
1.59kg/3.391b 

Total  weight  2.23kg/4.901b 

Power 

Standby 

Typical 

-10W  (without  input 
signal), 

~16W  (with  input  signals) 

Fully  programmed 

With  Aurora  links  active 

Memory 

Frame 

buffer/PPC 

Coeff. 

buffer 

DSP  buffer 

192  MByte  DDR 

SDRAM 

9  MByte  QDR  SRAM 
16Mbyte  SDRAM 

Upgrade  by  2x  possible 

Upgrade  by  2x  possible 

Ext. 

storage 

Compact  flash, 

Microdrive 

Embedded  Linux  storage 

Figure  11:  Summary  table  of  the  specifications  of  the  Image  Processing  Module  (IPM) 
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3.2.3  Image-Processing  Module  (IPM)  Packaging 

In  order  to  protect  the  IPM  PCB  board  and  the  sensitive  electronics,  a  metal  enclosure  made  out  of 
folded  steel  which  was  lasercut  precisely.  The  enclosure  also  guides  the  airflow  in  a  defined  channel 
limited  by  the  enclosure  and  therefore  improves  cooling  of  the  components  of  the  PCB  board, 
especially  the  VP70  FPGA,  the  TI  DSP  and  the  DC/DC  converters. 

Figure  12  and  Figure  13  show  photographs  of  the  packaged  Input/Output  modules  suitable  for  19 
inch  rack  mounting.  The  height  is  1U  or  1.75  inches.  The  input  and  the  output  Aurora  links  were 
configured  to  come  into  the  back  in  order  to  make  the  packaging  as  easy  as  possible.  The  assignment 
is  defined  in  the  firmware  as  follows:  on  the  IPM  on  the  transmitting  side  we  have  it  configured  as 
four  inputs  and  one  output;  on  the  IPM  at  the  receiving  side  it  is  configures  as  one  input  and  four 
outputs. 

The  IPM  has  back  the  same  compact  flash  interface  like  the  IOM  which  carries  the  FPGA 
configurations  and  could  be  used  in  the  future  even  to  run  embedded  Linux.  This  makes  upgrading 
the  firmware  easy  and  allows  storing  several  configurations  which  can  be  stored  on  the  flash  card. 

The  DVI  output  on  the  IPM  was  only  used  for  debugging  purposes  so  far  to  monitor  the  video  data 
stream. 


Figure  12:  Front  view  of  IPM  19”-rackmountable  1U  module  with  the  eight  auxiliary  high  speed 

interfaces  and  status  LEDs  in  the  front 


Figure  13:  Back  view  of  IPM  19”-rackmountable  1U  module  from  where  the  connections  to  the  IOM  and 
NIM  are  made  as  well  the  compact  flash  for  the  FPGA  configurations  is  being  inserted 
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3.3  Network  Interface  Module  (NIM) 


The  NIM  module  is  either  receiving  video  and  control  packets  from  the  IPM  on  the  transmitting 
side  or  transmitting  packets  to  the  IPM  on  the  receiving  side.  The  NIM  was  the  only  module  designed 
such  that  the  embedded  code  on  the  transmitting  and  receiving  side  are  identical.  The  NIM  takes  care 
of  the  networking  stack  of  the  “BlueEagle”  packet  protocol  on  UDP.  The  NIM  was  also  designed 
with  two  DVI  outputs  to  allow  future  use  as  remote  desktop  without  hard  drive  for  secure 
environments. 

3.3.1  Network  Interface  Module  (NIM)  Design 

Figure  14  The  NIM-2  module  has  512MByte  DDR  memory  on  board  and  can  accept  up  to  four 
Aurora  links  as  inputs  allowing  then  future  scaling  to  4x  HDTV  resolution.  Two  channels  were 
converted  to  2Gbps  Fibrechannel  to  allow  attachment  of  external  Fibrechannel  hard  drives. 

IEEE  13  94a  is  implemented  such  that  it  can  transfer  high  speed  data  up  to  400Mbps.  The  hardware 
supports  100/1000M  Ethernet.  The  chipscope  interface  was  implemented  the  same  way  as  on  the 
IPM. 
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Figure  14:  Network  Interface  Module  (NIM)  functional  block  diagram 

Figure  15  shows  a  photograph  of  the  assembled  NIM  PCB  with  its  1,658  parts. 

Altogether  there  are  84  unique  parts  on  the  board  and  the  PCB  has  an  overall  of  6,629  pins.  98%  of 
all  the  SMD  components  could  be  placed  automatically  by  our  pick&pace  tools  and  only  35  parts  had 
to  be  populated  by  hand  beside  all  the  through-hole  connectors.  The  board  is  16”  x  5.6”  in  size  and  is 
powered  as  the  IOM  by  standard  56 W  Thinkpad  certified  international  power  supplies 
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Figure  15:  Fully  populated  Network  Interface  Module  (NIM)  after  PCB  assembly 


3.3.2  Network  Interface  Module  (NIM)  Specifications 

Figure  16  shows  a  summary  of  the  NIM  specifications.  The  two  DVI  channels  can  be  configured 
either  in  dual  DVI  mode  using  both  connectors,  in  dual-link  DVI  mode  using  the  primary  connector 
or  single-link  DVI  using  the  primary  connector.  All  other  specifications  of  the  connections  are  similar 
to  the  IOM  and  IPM. 


Category 

Parameter 

Value 

Comment 

Inputs 

Infiniband 

600  Mb/s  to  3.125  Gb/s 

RocketIO,  reconfigurable 

Outputs 

Pimary 

DVI-D 

Seconary 

DVI-D 

Ethernet 

Infiniband 

Fibrechannel 

3.96  Gb/s  (max.  165  MHz  pixel 
CLK) 

3.96  Gb/s  (max.  165  MHz  pixel 
CLK) 

100/1000  Mb/s 

600  Mb/s  to  3.125  Gb/s 
lGb/s  or  2Gb/s 

Compliant  to  DVI  1 .0 

Compliant  to  DVI  1 .0 

RocketIO,  reconfigurable 

Control 

Interfaces 

RS232 

USB1.1 

IEEE  13  94a 

1 1 5kb/s 

12Mb/s 

400Mb/s 

Others: 

Keyboard 

Mouse 

Smartcard 

8-contact 

FPGA 

Embedded 

CPU 

Rocket-IO 

2x  Power 

PC 

Aurora/Inifiniband/F  ibrechannel 
Embedded  Linux  optional 

Infiniband  lx  connectors 

PC  Board 

16-layer 

16”  x  5.6”  x  0.093”,  0.68 
kg/1.59  lb 

Weight  fully  populated 

Chassis 

0.045”  steel 

16.5” x  5.75” x  1.5”,  1.47 
kg/3.47  lb 

Total  weight  2.15  kg/5. 061b 

Memory 

Frame 

buffer 

PowerPC 

512  MByte  DDR 

32  MByte  SDRAM 

Upgrade  by  2x  possible 

Ext. 

storage 

Compact  flash,  Microdrive 

Embedded  Linux  storage 

Figure  16:  Summary  table  of  specifications  of  the  Network  Interface  Module  (NIM) 
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3.3.3  Network  Interface  Module  (NIM)  Packaging 

In  order  to  protect  the  IPM  PCB  board  and  the  sensitive  electronics,  a  metal  enclosure  made  out  of 
folded  steel  which  was  lasercut  precisely.  The  enclosure  also  guides  the  airflow  in  a  defined  channel 
limited  by  the  enclosure  and  therefore  improves  cooling  of  the  components  of  the  PCB  board, 
especially  the  VP70  FPGA,  the  TI  DSP  and  the  DC/DC  converters. 

Figure  17  and  Figure  18  show  photographs  of  the  packaged  Input/Output  modules  suitable  for  19 
inch  rack  mounting.  The  height  is  1U  or  1.75  inches.  The  input  and  the  output  Aurora  links  were 
configured  to  come  into  the  back  in  order  to  make  the  packaging  as  easy  as  possible.  The  assignment 
is  defined  in  the  firmware  as  follows:  on  the  IPM  on  the  transmitting  side  we  have  it  configured  as 
four  inputs  and  one  output;  on  the  IPM  at  the  receiving  side  it  is  configures  as  one  input  and  four 
outputs. 

The  IPM  has  back  the  same  compact  flash  interface  like  the  IOM  which  carries  the  FPGA 
configurations  and  could  be  used  in  the  future  even  to  run  embedded  Linux.  This  makes  upgrading 
the  firmware  easy  and  allows  storing  several  configurations  which  can  be  stored  on  the  flash  card. 

So  far,  the  DVI  output  on  the  IPM  was  only  used  for  debugging  purposes  to  monitor  the  video  data 
stream. 


Figure  17:  Front  view  of  NIM  19”-rackmountable  1U  module  with  smart  card  reader 


B— pi 


Figure  18:  Back  view  of  NIM  19”-rackmountable  1U  module  with  dual  DVI  outputs  and  Ethernet  RJ45 

connector 
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3.4  Overall  Packaging  of  “BlueEagle”  System 

Figures  19  &  20  Rather  than  having  the  individual  modules  being  stacked  directly  on  top  of  each 
other  we  staggered  them  in  a  3U  high  desktop  rack.  This  also  allowed  us  to  integrate  the  power  strip 
and  the  Thinkpad  AC/DC  power  supplies  into  the  same  box  and  make  it  much  easier  to  move  the 
system  around  even  though  it  added  weight.  Certainly  it  allows  the  cables  to  be  plugged  in  securely 
with  a  lower  risk  to  slip  out.  Compared  to  the  enclosures  for  the  individual  modules,  this  box  was 
acquired  as  an  off-the-  shelf  item.  All  the  other  enclosures  were  custom  designed  for  each  module. 


Figure  19:  Fully  packaged  “BlueEagle”  transmitter  system  unit 


Figure  20:  Fully  packaged  “BlueEagle”  receiver  system  unit  using  identical  hardware  modules  as  in  the 
transmitter  system  unit  but  programmed  differently  to  accommodate  functionality 
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3.5  Cameras  and  Pan&Tilt&Zoom  (PTZ)  Units  used  with  “BlueEagle”  System 


A  few  box  camera  systems  with  appropriate  PTZ  units  were  considered  during  the  “BlueEagle” 
project.  The  following  mentions  three  systems  which  were  considered  in  the  BlueEagle  design 
accommodating  their  different  video  interfaces.  The  system  is  not  limited  using  other  camera 
systems.  They  could  be  accepted  with  no  or  minor  changes  if  the  physical  interfaces  are  identical. 

3.5.1  Ikegami  HDL-40  camera  with  Fujinon  Motorized  Lens  and  Pan&Tilt  System 

Figure  21  Prior  to  the  start  of  this  project,  IBM  Research  was  in  the  position  to  buy  with  capital 
money  a  high-  end  box  camera  system.  We  bought  an  Ikegami  ®  HDL-40  box  camera.  The  camera 
has  three  CCD  sensors,  each  at  1920x1080  resolutions,  and  offered  at  the  time  one  of  the  best  signal 
to  noise  rations.  It  transmits  the  video  signal  digitally  according  to  the  SMPTE  HD-SDI  standard  as  a 
1920x1080  format  at  60Hz  frame  rate  interlaced.  The  lens  attached  to  the  camera  is  a  high-end 
motorized  HD  lens  from  Fujinon®.  The  Pan&Tilt  unit  is  also  a  professional  unit  from  Fujinon  which 
is  weight  balanced  to  carry  camera  and  lens.  We  used  this  camera  system  as  a  reference  system  for 
the  “BlueEagle”  system  because  of  its  image  quality  and  sensitivity. 


Figure  21:  Ikegami  HDL-40  camera  with  Funjinon  motorized  HD  lens  and  Funjinon  Pan&Tilt  system 


3.5.2  Silicon  Image  Camera  with  ESI  Pan&Tilt  and  Computar  motorized  lens 

A  more  cost-effective  solution  by  a  factor  of  about  8  is  the  combination  of  a  Silicon  Image  SI- 
1920HD-RGB  camera,  Computar  M10Z1 1 18MSP  motorized  lens  and  ESI  DPT-1 15  Pan/Tilt  head. 
Figure  22  and  Figure  23  show  the  camera  module  with  Computar  motorized  lens  and  its  integration 
with  the  ESI  Pan/Tilt  unit,  respectively. 
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The  sensor  used  in  the  Silicon  Imaging  SI-1920HD-RGB  2/3”  format  camera  is  a  single  chip  CMOS 
sensor  from  Rockwell  with  a  bayer  color  pattern.  The  camera  interface  is  CameraLinkTM  and  the 
sensor  can  be  read  out  up  to  60Hz  frame  rate  at  1920x1080  pixel  resolution.  However,  in  our 
experiments  we  used  the  sensor  at  the  same  video  timing  as  on  the  Ikegami  camera  This  allowed  for 
longer  integration  time  and  for  a  better  signal  to  noise  ratio.  Obviously,  the  Ikegami  HD  camera  is 
still  superior  in  all  respects.  But  it  can  be  anticipated  that  CMOS  sensors  will  catch  up  and  offer  other 
helpful  features,  such  as  windowing  which  allows  to  read  out  section  of  the  sensor  at  higher  speed. 


Figure  22:  SI-1920HD-RGB  Camera  Mounted  on  Computar  H10Z1 1 18MSP  Motorized  2/3”  Lens 


Figure  23:  ESI  DPT-115  Pan/Tilt  w/  Computar  H10Z1118MSP  Lens  w/  SI-1920HD-RGB  Camera 
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3.5.3  ISG  camera 


The  “BlueEagle”  system  is  prepared  to  be  connected  to  the  prototype  camera  from  ISG  which 
provides  four  DVI  channels  simultaneously.  The  ISG  camera  is  a  single  chip  CMOS  camera  with 
3840x2160  pixels  and  a  bayer  color  filter  pattern  on  top  of  the  sensor.  Multiple  channels  can  be 
handled  by  the  “BlueEagle”  system  by  streaming  the  signals  through  modules  in  parallel  and  share 
boundary  data  through  the  auxiliary  Aurora  channels  on  the  Modules  (see  Figure  24) 
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Figure  24:  Interconnections  between  scaled-up  “BlueEagle”  system  to  accommodate  for  example 
transmission  of  digital  signale  of  4x  HDTV  resolution  (3840x2160) 
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To  the  T221  monitor 


3.6  “BlueEagle”  System  in  a  Surveillance  Application  with  IBM’s  Peoplevision 
Software 

Figure  25  an  implementation  of  the  “BlueEagle”  system  in  a  surveillance  application  together  with 
the  PeopleVision  software  from  IBM  Research  which  is  selecting  the  location  and  size  of  the  region 
of  interest  on  the  transmitting  side  near  the  camera.  It  is  the  configuration  how  the  systems  are 
installed  at  two  US  Army  sites.  In  case  of  the  site  installations,  because  of  the  close  proximity  of 
monitor  and  camera,  we  also  allow  a  so-called  split  screen  mode  in  which  the  T221  shows  on  the 
right  side  the  original  camera  signal  converted  to  DVI  and  the  left  side  the  receiving  image  going 
through  the  whole  BlueEagle  system. 


Figure  25:  “BlueEagle”  system  block  diagram  with  IBM  PeopleVision  surveillance  software 
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4.  Software  for  the  “BlueEagle”  System 

4.1  Graphical  User  Interface  (GUI) 

The  WindowBE  program  was  designed  to  both  provide  an  interface  for  visualizing  the  region  of 
interest  (ROI)  and  background  images  provided  by  the  IOM  and  allowing  the  user  to  dynamically 
change  both  the  parameters  of  the  IOM  and  the  image  source  through  panning,  tilting,  zooming 
and/or  focusing  the  camera.  In  addition,  when  connected  to  the  SI  image  camera  and  ESI  stage,  the 
current  deliverable  system,  defining  and  using  camera  presets  along  with  iris  control  are  also 
supported.  By  using  a  user  selectable  chroma-key  the  IOM  replaces  the  pixels  which  match  the 
chroma-key  with  appropriate  image  data. 

The  WindowBE  program  is  a  32  bit  windows  program,  which  means  that  it  will  run  on  any 
Windows  95  and  later  Operating  System.  However,  since  it  is  optimally  used  with  a  wheel  mouse  and 
the  wheel  mouse  was  only  introduced  in  Windows  2000,  it  is  recommended  that  Windows  2000  or 
later  Operating  System  be  used. 

The  WindowBE  program  is  designed  to  run  in  stand  alone  mode  such  that  its  entire  input  is 
manually  controlled  by  the  user.  Along  with  the  program  a  text  file  WindowBE.ini  is  also  located  in 
the  same  directory  as  the  program.  If  necessary,  as  described  later  in  this  document,  this  file  can  be 
modified  with  any  text  editor.  In  addition,  it  is  designed  to  be  run  in  conjunction  with  the 
PeopleVision  software.  When  run  with  the  PeopleVision  software,  the  PeopleVision  software 
automatically  loads  the  WindowBE  program.  The  PeopleVision  software  can  automatically  control 
the  position  and  size  of  the  ROI  by  directly  communicating  with  the  WindowBE  program.  When  the 
PeopleVision  software  is  closed  down  the  WindowBE  program  is  also  automatically  unloaded. 


4.2  Screen  Layout 

Three  windows  are  presented  by  the  WindowBE  program,  a  ROI  window,  a  background  window, 
and  a  control  window  as  shown  in  Figure  26.  The  chroma-key  has  been  selected  to  be  magenta,  for 
this  illustration. 

As  shown  in  Figure  26  by  the  magenta  colored  areas,  the  ROI  and  background  windows  are 
superimposed  on  top  of  each  other.  The  entire  resolution  of  the  HDD  camera,  which  is  1920x1080 
pixels,  is  comprised  by  these  windows.  The  remaining  bottom  portion  of  the  screen  (i.e.  1920x120 
pixels)  is  used  for  the  user  control  window. 

Along  with  presenting  information  about  the  ROI,  the  control  window  allows  for  controlling  many 
hardware  parameters.  In  particular,  the  frequency  of  updating  the  background,  along  with 
constraining  the  allowed  background  bandwidth,  can  be  set  to  a  multiplicity  of  values  with  slider  bars. 
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Figure  26:  The  WindowBE  user  interface 


Several  different  user  interface  parameters,  like  linking  the  ROI  movement  to  mouse  movement, 
toggling  on  and  off  the  ROI  border,  along  with  allowing  manual  or  People  Vision  control  of  ROI 
movement,  can  be  set.  In  addition,  Pan/Tilt/Zoom  and  Focus  of  the  camera  is  supported.  Finally,  a 
dialog  box  to  control  setting/going  to  Preset  camera  position,  and  looping  through  preset  camera 
positions  at  selectable  time  increments  is  supported. 
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5.  Utilities  Developed  in  Support  of  “BlueEagle”  System 

5.1  Utility  Software  for  Control  of  Silicon  Imaging  single  CMOS  Sensor  HD  Camera 

A  standalone  Windows  application  program  ESI_PanT.EXE  has  been  written  to  integrate  the  control  of 
both  asynchronous  ports. 

This  software  consists  of  a  simulated  joystick  control  for  the  pan  and  tilt  functions  of  the  camera 
positioning  system,  as  well  as  buttons  for  sending  individual  pan  and  tilt  commands  at  user-selected 
speeds  and  choice  of  axes.  There  are  sixteen  preset  camera  positions  which  may  be  set  which  also 
include  lens  zoom  and  focus  settings.  Lens  iris  control  cannot  be  preset  and  can  only  be  controlled  by  its 
slider  function.  There  also  are  slider  functions  for  adjusting  lens  zoom  and  focus.  There  is  an  AUTO  Pan 
mode  which  permits  continuous  panning  of  up  to  all  sixteen  preset  pan/tilt  camera  positions,  with  user- 
selected  hold  times  at  each  position  of  from  8  to  60  seconds. 

In  order  to  store  a  camera  preset  position,  use  the  joystick  control  to  position  the  pan/tilt  head  at  the 
desired  location,  then  use  the  zoom  and  focus  sliders  to  adjust  the  zoom  and  focus.  Click  on  the  ‘GoTo 
Preset’  box  which  will  then  be  labeled  ‘Store  Preset’  and  then  click  on  the  desired  ‘Preset  n’  button.  The 
‘Store  Preset’  label  will  be  automatically  changed  back  to  ‘GoTo  Preset’.  The  new  preset  information  is 
now  stored. 

To  go  immediately  to  any  stored  preset  position,  simply  click  on  the  desired  ‘Preset  n’  button. 

When  first  powering  up  the  system  or  loading  the  application  program,  use  the  ‘Init’  button  to  initialize 
the  basic  settings  for  the  SI-1920HD-RGB  camera.  Pressing  this  button  sends  a  series  of  camera 
commands  which  will  first  check  that  the  camera  is  connected  and  powered  on,  and  will  then  set  up  the 
proper  settings  for  color  and  the  proper  clock  frequency  for  1920  x  1080  @  60Hz  operation.  There  is  a 
window  where  the  camera  responses  for  each  command  will  be  displayed  for  the  user. 

The  camera  contains  a  set  of  default  settings  which  consists  of  all  42  of  the  SI-1920HD-RGB  camera 
registers.  There  are  also  three  Preset  storage  locations  (Preset  1,  Preset  2,  and  Preset  3)  which  allow  the 
user  to  store  a  complete  set  of  camera  registers  which  have  been  altered  by  the  user  to  adjust  them  for  a 
specific  camera  application.  The  camera  will  boot  up  whenever  its  power  is  cycled  off  and  then  on. 

The  register  set  that  the  camera  loads  upon  boot-up  will  depend  upon  previous  setup  commands  issued  by 
the  user.  For  example,  if  an  “ldl”  command  has  been  issued  by  the  user,  then  the  camera  will  boot  up  to 
the  register  set  which  the  user  has  saved  in  the  Preset  1  register  storage  location.  This  is  the  way  that  the 
camera  can  return  to  normal  operation  using  the  user’s  specific  register  settings  for  a  previously  set-up 
camera  application. 

Should  there  be  any  problem  which  might  be  associated  with  bad  register  data  in  one  of  the  Preset 
storage  locations,  then  the  recovery  procedure  would  be  to  issue  an  “ldO”  camera  command  to  force  the 
camera  to  boot  up  with  the  manufacturer’s  default  settings,  and  then  power  cycle  the  camera.  Then  the 
user  can  run  the  ‘Initialize’  command  from  the  GUI  which  will  update  several  of  the  registers  to  the  user’s 
specific  application.  Finally,  this  restored  set  of  registers  can  be  copied  into  the  selected  Preset  location 
(e.g.  by  issuing  an  “lei”  camera  command).  Now  it  is  possible  to  issue  an  “ldl”  camera  command  which 
sets  the  camera  up  to  boot  up  from  Preset  1  at  the  next  camera  power  off/on  cycling. 

It  is  also  possible  to  load  the  camera  registers  from  the  Preset  storage  locations  by  issuing  the  simple  one- 
character  commands  (i.e.  issue  “1”  to  load  from  Preset  1,  “2”  to  load  from  Preset  2,  and  “3”  to  load  from 
Preset  3).  This  feature  provides  the  flexibility  of  switching  between  several  sets  of  camera  settings  in 
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order  to  quickly  adjust  to  different  lighting  conditions,  for  example,  or  perhaps  to  simply  make  visual 
comparisons  between  settings. 

Initialization  Camera  Commands:  (Executed  when  GUI  ‘Initialize’  button  is  pressed) 


"ldl" 

"lyf40032" 

“Iy7000c0” 

“lg766690” 

"Ic34ae05" 

or 

"Ic34d20f' 

“lei” 


Get  Camera  Status 

Select  Preset  No.  1  Settings  for  Camera  Bootup 

Set  Reg  f4H  for  Camera  Individual  Analog  Gain  Controls  for  RGB  Mode 
Set  Reg  70H  Bit  7  to  Enable  Gamma  Curve 

Set  RGB  Analog  Gains  (B=7,  Gl=6,  G2=6,  R=6),  4  Black  Offsets  to  90H 
Set  Camera  Clock  Frequency  to  75.0  MHz  (for  1920  x  1080  @  60  Hz) 

Set  Camera  Clock  Frequency  to  37.5  MHz  (for  1920  x  1080  @  30  Hz) 

Save  all  camera  register  values  into  Preset  1  location 

Get  Camera  Status  (leaves  camera  status  response  on  GUI  display) 


Individual  camera  control  commands  may  be  selected  from  a  list  or  written  explicitly  into  the  ‘Camera 
CMD’  text  box  and  then  clicking  on  the  ‘SEND  Camera  CMD’  button. 

Figure  27  shows  a  screen  capture  of  the  utility  program  for  the  camera  sensor  and  the  PTZ  unit  attached 
to  it. 
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Figure  27:  Screen  capture  of  the  Utility  Program  Manually  controlling  the  CameraLink  Cameras 
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5.2  Video  Server  to  Capture  Uncompressed  Video  Data  in  Real  Time 

A  standard  IBM  e-server  was  acquired  independent  of  this  contract  and  specialized  software  was 
written  which  will  record  and  play  back  HD  YCrCb  4:2:2  data  at  30  Hz  interlaced.  The  server  was 
modified  to  include  dual  processor  2.66  GHz  CPUs  and  a  SCSI  RAID  system  consisting  of  4  SCSI 
disks  with  a  total  disk  capacity  of  136  GB.  A  20  minute  movie  section  and  be  recorded  and  played 
back  without  skipping  a  single  frame.  We  utilized  a  PCI  video  capture  card  from  DVS,  which  we  had 
acquired  prior  to  this  contract,  to  record  and  play  back  the  data.  To  capture  uncompressed  video  at 
the  above  resolution,  the  server  must  store  data  at  a  sustained  rate  of  over  120  MB/second.  It  was 
ascertained  empirically  that  storing  data  through  an  Operating  System  file  system  was  much  too  slow 
for  the  above  task,  so  that  we  needed  to  directly  write  to  the  disk  on  a  sector  by  sector  basis.  Since 
directly  writing  to  the  disk  can  easily  corrupt  the  Operating  system,  much  caution  and  checking  was 
built  into  the  software.  In  addition,  the  dual  processor  architecture  was  taken  advantage  of  by  writing 
multi-threaded  software  such  that  the  load  on  both  CPUs  was  balanced.  In  addition,  for  debug 
purposes,  a  helper  program  was  written  which  converts  and  displays  the  YCrCb  data  into  RGB  format 
on  a  frame  by  frame  basis. 

The  video  server  allows  us  to  do  a  detailed  analysis  of  the  quality  of  compression  in  our  ROI 
system.  In  addition,  it  will  enable  storing  and  retrieving  clips  such  that  our  hardware  and  the 
associated  PeopleVision  software  can  be  tested,  demoed,  and  improved  by  replaying  the  same  test 
sequences  again  and  again.  We  also  purchased  removable  hard  drive  cartridges  to  be  able  to  archive 
desired  video  sequences. 

5.3  Digital  Packet  Video  Link  Protocol 


During  the  course  of  this  contract  we  were  in  parallel  active  in  the  VESA  standards  and  were 
spearheading  the  effort  to  make  the  Digital  Packet  Video  Link  (DPVL)  into  an  industry  standard.  The 
committee  was  launched  under  IBM’s  leadership  on  April  24,  2002  and  the  standard  was  approved  on 
April  18,  2004.  Figure  28  shows  the  packet  definition  of  the  basic  header  under  DPVL  and  Figure 
29  shows  the  timing  format  of  the  packets  under  the  DPVL  standard. 
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Origin  of  the  Virtual 


Figure  29:  Video  data  and  packet  timing  format  in  the  DPVL  protocol 


More  information  about  DPVL  can  be  downloaded  from  the  VESA  website,  specifically  the  90- 
page  standard  document  describing  the  DPVL  protocol  in  all  detail. 

Figure  30  illustrated  the  use  of  the  IOM  and  the  NIM  from  the  “BlueEagle”  system  for  prototyping 
solutions  using  the  DPVL  protocol  and  future  packet  protocols  over  IP. 
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Figure  30:  Example  how  BlueEagle  IOM  and  NIM  are  being  used  for  DPVL  prototyping 
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6.  Delivery  of  Two  “BlueEagle”  Systems 

The  first  “BlueEagle”  system  was  hand  delivered  to  ARL  in  Adelphi  on  June  8,  2005.  The  system 
was  setup  as  it  is  illustrated  in  Figure  25.  Figure  31  and  Figure  32  show  the  system  setup  on  an 
optical  bench  in  the  ARL  lab.  On  the  left  side  is  the  transmitting  side  (Tx)  of  the  system  and  on  the 
right  side  the  receiving  side  (Rx)  with  an  IBM  T221  monitor.  The  Silicon  Imaging  3Mpixel  CMOS 
camera  with  its  PTZ  unit  is  mounted  on  the  top  of  the  shelf  on  the  left  side.  The  communication 
between  the  two  sides  of  the  system  is  being  entirely  handled  over  the  internal  ARL  100Mbit- 
Ethemet  network.  The  LCD  monitor  on  the  left  side  is  to  the  remote  computer  (see  Figure  25,  PC  not 
visible  in  picture)  which  contains  the  Peoplevision  application  and  also  shows  the  output  of  the  down 
sampled  image  which  is  fed  into  the  remote  computer.  The  IBM  PC  below  the  T221  contains  the 
“BlueEagle”  GUI  application  and  allows  overall  system  control. 


Figure  31:  Photo  of  the  “BlueEagle”  prototype  system  setup  on  the  optical  bench  at  ARL 

Due  to  the  proximity  of  the  “BlueEagle”  transmitter  and  receiver  we  decided  to  show  the  system  in  a 
so  called  ‘split  screen’  mode.  Figure  33  is  a  photo  of  the  T221  screen  showing  on  the  left  side  the 
captured  signal  at  the  DVI  output  (1920x1200)  of  the  receiving  side  of  the  system  and  the  right  side 
the  CameraLink  signal  to  DVI  from  the  IOM  of  the  transmitting  side.  T221  was  programmed  in  this 
mode  to  accept  asynchronous  video  signals.  The  left  side  acts  as  a  master  channel  to  which  the  right 
side  is  being  synchronized  to  (by  switching  the  cables  to  the  T221  one  could  invert  that  sequence). 
For  this  purpose  we  inserted  at  the  output  of  the  Tx  IOM  a  DVI  splitter  which  allows  to  replicate  the 
DVI  data  signal  and  take  it  to  the  secondary  channel  of  the  T221.  This  mode  allows  the  simultaneous 
viewing  of  both  video  streams  on  a  single  display  device  in  full  resolution  and  allow  us  to  evaluate 
system  behavior.  A  second  “BlueEagle”  system  was  setup  a  month  later  on  July  1 1  in  Natick  in  a 
similar  configuration. 
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Figure  32:  Photo  of  the  “BlueEagle”  prototype  system  setup  on  the  optical  bench  at  ARL 


Figure  33:  Photo  of  the  T221  screen  showing  on  the  left  the  1920x1200  received  image  on  the  receiving 
side  of  the  system  and  on  the  right  the  1920x1080  image  captured  from  camera  converted  to  DVI  format 


7.  Conclusions 


The  “BlueEagle”  system  created  under  this  contract  is  a  powerful  prototype  system  which  allows 
running  experiments  in  different  visualization  scenarios  primarily  from  live  high-resolution  camera 
feeds.  It  has  been  demonstrated  that  the  system  can  be  adapted  to  different  situations  without 
changing  any  hardware.  Building  the  “BlueEagle”  system  in  a  modular  way  was  not  only  highly 
beneficial  from  a  budget  point  of  view  but  also  from  a  risk  management  point  of  view  because  each 
of  the  three  new  modules  (IOM,  IPM  and  NIM)  designed  under  this  contract  have  the  complexity  of  a 
PC  motherboard.  Further  miniaturization  is  certainly  possible  with  the  latest  FPGA  chips  coming  out 
from  Xilinx  later  this  year  but  it  was  not  in  our  budget  to  pursue  that  at  this  point. 

The  modular  design  of  the  system  opens  up  opportunities  to  use  individual  modules  for  other 
purposes.  We  use  for  example  the  IOM  for  a  potential  baseband  implementation  on  the  60GHz 
wireless  range  for  wireless  high  image  content  video  transmission.  Currently  the  combination  of  IOM 
and  NIM  is  in  use  by  a  summer  student  who  prototypes  an  implementation  of  DPVL.  Biggest  issue 
right  now  is  the  limitation  of  resources  for  writing  embedded  code  for  the  system. 

Two  “BlueEagle”  systems  have  been  setup  in  two  different  locations,  one  at  ARL  in  Adelphi  and  one 
at  SSC  in  Natick.  Throughout  the  remainder  of  the  year  we  plan  to  have  a  few  more  upgrades  to  the 
deployed  BlueEagle  system  and  we  might  use  a  follow-on  of  the  IOM  together  with  a  board  which 
contains  a  cell  processor  for  high-end  imaging  and  video  surveillance  applications.  Still,  in  both  cases 
a  lot  of  time  has  to  be  spent  to  write  the  embedded  respective  code. 
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